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Chapter 12 



COMPOSITION OF USER AND CONTROL GROUPS AND THE 
INFORMATION USE HABITS OF USERS AND CONTROLS . 



One of the major aims of the SDI Inves tigatj on* was to 
discover the effect that provision of an SDI service might 
have oji the use made of information by recipients of the 
s e r v i c o . 

The method adopted was that of a ques tionnairc (Appendix 
12A) sent to participants in the Investigation before 
the start of the SDI service to establish their use of 
xnformation at the time. This same questionnaire was 
again sent to participants when they had boon receiving 
sorvi.ee for some 12 months. It was hoped in this way to 
establish the pattern of information use before and after 
the service and therefore to ascertain the changes during 
that period. Since the questionnaire was also on each 
occasion sent to a comparable sample of people who did 
not receive the service, it was hoped that it would be 
possible to isolate those changes which were unique to 
the recipients of the service and thus, presumably, due 
to the effect of the SDI Service. 
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Ill ordoj, to remove avoidable bias in the answers,' the 
f irst quos tionnairo vas sent; to a random sample of elec- 
tron J rs rosearch workers and only after return of the 
quos tionnairo wore the respondents assigned to cither the 
User or Control group. Similarly no mention vas made of 
the SDI Investigation .in the covering: letter (Appendix 
12B) which purported to be a general survey of information 
use by olectroni.es research workers and was signed by 
Mr. T. M. Aitclii-son in his capacity as Manager, Information 
Research Tor INSPBC. 

It should .bo pointed out that all communiea lions relating 
to SDT throughout the In v«s fcigat i.on were signed by Mr. P. 
Claguo as Manager of the SDI Invos t Iga t:i on . 

After return of the quos tionnairo those peopj o vho were 
assigned to the user group were sent a letter inviting 
them to take part /In the Investigation. Those assigned 
to the Control Group received no further communication 
-until the questionnaire was repeated up to two and a half 
years later. 

The period during which the first questionnaire was 
distributed lasted approximately from June 1967 to Apr:M 1 968 . 
This rather long period is explained by the fact that 
collection of the required user and control groups took 
a Jong time and it was icontinualiy necessary to approach 
new people. 

Because of this long period, and the fact that enrolling 

of respondents into the user group went on at the same 

time, Jt is possible that the Jink between the ques tionnairo 

and the SDI Investigation may have become apparent, but 

it is not considered likely that this would have affected the 

sELmpie seriously . 

The 'questionnaire contained 22 questions, the first six of 
which were intended to discover the characteristics of 
the respondent e.g. information on his age, qualifications, 
status, type of research done, etc. The next seven questions 
(Nos 7-1 J ) were intended to discover some tiling of the 
respondent's attitude to published information e.g. whether 
he thought published information important to himj whether 
he considered he was adequately informed, and what problems 
he encountered in keeping abreast of information. 
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The last series of questions (Nos 12-21 ) were concerned 
with the respondents use of libraries, technical and 
scientific periodicals, and abstracting and indexing 
services, and the amount of time spent in information 
ga thering. 

A few of the questions were open-ended. The coding 
categories for most of these are shown in the results 
in Appendix 12C. however the categories for questions 
3 and h arc given Ln Figure 1 below. 

Figure 1 » Coding of open-ended questions 

Question No . 3 Position held by respondent 

Category 1. Section Enginf^r ; Team Member; Design 
& Development Engineer; Physicist; 
Application Ba^inccr: Research Engineer: 
Research Physicist; Systems Engineer. 

Category 2. Principal Scientific Staff;. Senior 
Scientist; Senior Engineer; Senior 
Physicist; Senior Test Engineer. 

Category 3. Leading Scientific Staff; Group Leaders; 

Section Leaders; Leading Staff; Project 
Loader; Team Leader; Section Head. 

Category k. Department Head; Chief Engineer; 

Assistant Chief Engineer; Director; 
Chief Physicist; Chief Scientist; Chief 
Designer; Manager; Consultant: Advisory. 

Question No ♦ h Subject of research 

1. Microwave theory and techniques including microwave 
tubes and microwave semiconductor devices. 

r 

2. Quantum electronics, quantum optics, lasers, 
laser applications • 

3 • Semiconductors and mrcroclcc tronics . 
k. Radar systems and equipment. 
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r '" 1 ». »••".. ■ I..,..,,,, 

'" ' A " t """" S '-<"»<>•".>„, .to,,,,,,,,,,,.,,., ,..„,„,„„„.,„. • 

7., PLasinas, ,r;i S cUsjelinreoH. 

8. Eloctron . pMo „ oll , ctro „ ^ ^ 

9- C„„„ ul ,. rs , (lat(1 , )roo , !S3jMe) imttorii 1 . ( , cocili tIon _ 
10. ControJ theory and sys t,.,„ s . 

"' .^iconduc'tort')?"^' 3 °" <1 ""™ («.l«d*»« 

12. Miscellaneous (onvironm«.tifr..i 

testing weapon. S^'^.p"^^ a«l 

13. Not stated by respondent. 
Results 

The originaJ number or poodIo in m 

groups who returned tho firs J 'f ' US °-' U " d confc rol 
-CoLy 1200 co m p risin , JS^^^S SHSr" 6 - 

replies. In the case of t e " ° C Ul ° ,; ' 00 Hi 

sent to the /, 50 or so X , ^o.tionnairo. were 

and from these 3 , 9 replios vJro received. ° y partici P a fci »tf 
In tlio case of tho mnfhAi 

all vh. ,,ad ropx^^^i '«»j; d "^ai~ ««. ro so „ t to 
received. ° uiy anu 280 replies were 

'Since, oving to pressure of work ,,n r n 
sent to chase non-replies th I follow-up letter was 
particularly for tho control, Percentage return was C ood, 
contact with them ftr^lT^SS b °° n »° 

t0 »"* — ^nnalres 
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"Howti in A,,, , I r /'"' mt questionnaire are 

uint^J for the sump |.(. ^ ' L ' " ! J"« sfcj «» are tab- 
and industry sub-groups pf oaclu V ' <' r ° v < ! ™«»<< 

4* the Appendices ar« s as foilovs:- &tvrn 

XnrormatLon Use fe, , j 0<mil , _ g , ^ 

Table No r , Co^Uion or sampLe of ^ ?imJ 

2 Composition or sample by qualifications. 

3 Composition of sample by position or status 
'« Composition or sample by research role. 

5 Compaction or sample by subject of research 
- Cor codxn 6 see Pi^re i above. OSoai <-» 

6 Composition of sample by type of research. 

9 -^t^tS^r" 0011 ^ ~ il " ^covering 

10 Difficulty in obtaining material. 

11 Difficulty in rinding time to read material 

™ TlltT^T^ iS midQ by Us ^ s *<> carry out 
a literature search. ' uv 

13 How confident are iiieinb«»rs «r n, 

they do keep informed 1. 8M,pJa t,,at 
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I'l Typos or library used by the sn,„plo 

13 Fronuc-noy of use of tho library service 

16 Location of the library usod. 

" sZal. 0t ParLoM "^ >■««' n^tarty „y tl ,o 

.8 Technical, periodicals considered ,„o S , ,„ rlo „ t 

19 Abstracts journals us.-d. 

20 Ti,„„ spent 8 cum,i„ e or reading technical 

21 2 n : iWt "•*"« — ««*-«««»« toobnical infer,,, 
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THESAURUS DEVELOPMENT AND DOCUMENT INDEXING 



Work on the Invos tigation began not long after the publica- 
tion of the first edition of the Engineering Joint Council's 
Thesaurus of Engineering Terms. It was natural, therefore, 
to seek to use this as the controlled vocabulary for document 
indexing and profile compilation. However, it was quickly 
apparent that the EJC Thesaurus was not an adequate tool 
for indexing the very detailed subject matter of documents 
in the field of electronics research and that it would be 
necessary to supplement it considerably. As a result it was 
decided to construct a thesaurus on cards to cover our 
particular field of interest and to incorporate into it such 
terms from the EJC Thesaurus as met our requirements. The 
same system for Narrower Term (NT), Broader Term (BT) , USE 
and Used for (UF) references was adopted. Related Term (RT) 
reference was not, in general, adopted: It was felt that 
proliferation of RT references, whose precise relation to the 
term in question was unclear, would be time-wasting and 
unhelpful and it was hoped to restrict the use of such cross- 
references to terms actually found linked in profiles. This 
was,, in fact, never done, and the types* of cross-reference 
used in the thesaurus were kept to NT, BT, USE AND UF. 

Addition of terms to the thesaurus was determined by the 
requirements of the documents ""which had to be indexed and 
of the information requirements from which subject profiles 
had to be compiled. The major concern was to expand the 
thesaurus to cope with these requirements as they arose and 
to fit new terms into the existing structure. This 
naturally resulted in a somewhat uneven growth, depending 
on the areas of particular concern to the Investigation and 
little attention was paid to any potential use it might have 
as a general-purpose tool for indexing material in the 
general field of electronics. 
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In compiling the thesaurus we were fortunate to have access 
to a draft thesaurus developed by Jean Aitchison in connection 
wi tli a rov:Ls ion of fcho printed indexes to Electrical and 
El oc Ironies Abstracts. 

Pro- coord i.nation of terms 

The major problem, as with all thesaurus construction, was to 
decide the optimum decree of prc-coordination of terms. The 
aim to produce the smallest thesaurus which would allow 
the subjects, of interest to each user to be distinguished 
from those in which he had no interest. Thus, if every user 
interested in transistors was interested in all "types of 
transistor, tb<in inclusion of the species of transistor in 
the thesauri^ was not necessary. However, if any user was 
interested in a particular species that species had to be 
capable of being distinguished in the thesaurus. 

In- general, as terms were required for document indexing or 
for profile compilation, they were added to the thesaurus 
after a check that no suitable term was already present in 
the thesaurus. Some attempt was made to control the 
proliferation of terms that would have resulted from the 
inclusion of highly-prc-co-ordinated terms. This was done 
by examining each candidate term to discover whether it 
could be adequately represented by a combination of two or 
more existing broader terms. The test for adequacy was that 
a document dealing with the subject of the specific term 
should provide useful information to a person interested in 
the subject of one or other of the general terms used in 
its place. Thus, the term "Colour Television Receivers" 
could be represented by the two terms "Colour Television" 
and "Television Receivers" since someone interested in colour 
TV or TV receivers would also be interested in colour TV 
receivers. On the other hand, in the case of Gallium Arsenide 
Lasers it was thought preferable to keep this as a single 
term since someoiae interested in gallium arsenide would 
probably not be interested in the many articles describing 
gallium arsenide lasers. 

This fairly simple rulc-of- thumb helped to restrict the 
number of individual used terms in the thesaurus. However, 
at least in tha early stages of thesaurus building, there 
was a tendency to use single terms where later experience 
(and the rule) would have dictated a greater degree of 
pre-co-ordination. Thus, initially, a concept such as 
capacitance measurement was dealt vith by the use of two 
single terms "Capacitance" and "Measurement". The liability 
of this practice to cause false coordinations in the searches 
soon became apparent and as a result there began a tendency 
to prefer the more specific terms. 
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Specificity 



This tendency was reinforced by the practice of allowing 
the stated informs ion requirements of users to dictate 
the level of specificity of "terms. That is to say that, 
unless some obvious advantage could be seen, it appeared 
best to adopt terms which directly expressed the real 
requirements of users rather than to produce an unnecessary 
complication by generalising or translating the terms. 
This, of course, does not affect the question of synonym 
control. 

The tendency to include very specific terms in the thesaurus 
did, however, considerably increase the number of terms in 
the thesaurus and consequently thte number required in 
profiles. This brought trouble in that the system had been 
designed for profiles having a maximum size of approximately 
one hundred terms. The considerable increase of specific 
terms which might otherwise have been dealt with by coordina- 
tion of two or more existing terms increased the number of 
terms required and in some cases the limits were exceeded. 

This situation was resolved by adopting «posting-up« in the 
indexing of documents, ie assigning the general term as well 
as the specific term in certain cases. This was done for 
such terms as Filters, Amplifiers, etc. Thus, where a user 
was interested in all types of filters his profile needed to 
use only the term Filters whereas before posting-up was 
adopted all the specific types of filter had to be included 
in the profile. (The 'posting-up' terms are listed on 
page 5). 

Growth of the thesa urus 

The maximum size of thesaurus estimated for in designing the 
computer system was *4,000 terms, not including lead-in terms. 

As expected the growth was very rapid in the early stages of 
profile compilation and indexing of the test document 
collection. Thus, after only four months the thesaurus 
contained some 1,500 terms and this figure had risen to some 
2,600 terms after ten months. By this time virtually all of 
the 600 profiles had been compiled and approximately 600 
documents had been indexed. Figures 1 and 2 show the growth 
of the thesaurus in the period July 1967 to , June 1969. 

From November 1968 the addition of terms is almost entirely 
due to the demands of document indexing and it is noticeable 
that in the following six months some 250 terms are added, 
ie approximately ten per week. This rate of growth may be 
higher than in other 'areas since many of the terms represent 
new solid-state materials being synthesized or investigated 
for their electronic properties. 



As a result when failure analysis was performed it was found 
that relatively few failures could be attributed dirJctw ?o 
the thesaurus In fact only eight percent of Rec t d 
Precision failures were so attributed. 

lS e |-S 0 ?n Perf0 r! ,anC S ° f the SDI Thesa ^us was also confirmed 

Sports ?S%^d e 7o/nPr t i° n ° f IndeX Lan ^ g es study (INSPEC 
reports 70/0, and 70/02) where the SDI controlled lanma^, 
was found to perform better than any of the other iSS * 
languages which were studied. X 

Document indexing 

Documents were indexed by assigning to them a set of appropriate 

r J P t0 " s <^cted from the thesaurus. The procedures £ 
scan the document or the significant portions of it e^ lilll 
abstract introduction, conclusion anS references and fo iJo^te 

ArthirsS^L 001106 ^ 3 With Wh±Ch the -thorns" n r 3 6 
At this stage the aim was only to record the significant concord 

The natural language terms were recorded on the left-hand sid P n r 
l^r 3 termr rk ^ ^ ^ Subse <^t operation ?he natural 
tSSSSs T Sr tranS l ate i into the controlled terms of the 
terms lel'e aJreaSv . of cas » s the appropriate controlled 

nl TLh a i read y Present in the thesaurus with cross-references 
or lead-ins from the natural- language terms. In many cases of 

ia^ical 116 Shere^^e^^,^ ^^"--languag.^. 
identical. Where the concept was catered for by an existing 

lo m / he a ^ r °P ria te term would be entered SwS'on 

number wrSVT* ^tt (F±gUre 5) a "d the corresponding 
S (?W I, ", he right- hand side of the document indexing 

^^[uraT-langu^el^m^* ^ *~ ™ d *> -present" 

Where the appropriate term to represent a natural- language 
concept was not specified in the thesaurus, the indJxfr" Sote 
opposite it on the right-hand side of the Index sheeT^he 

SS* te S rm T^T^ *° ^ th ™* to eft" for the 

new term. The document, document input sheet and indexing 
sheet were then passed to the thesaurus controller who 
approved the new entry or discussed it further, entered the 
appropriate term in the document input sheet and arranged for 
the thesaurus and the computer descriptor file to be updated" 

Copies of specimen sheets from the Descriptor File are given 
in Appendix 1 3 A. Author names and journal Coden also applar 
as descriptors in this list. pp 
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All »;ho indexing was done at the level of specificity of the 
document, e.g. an article on the deterioration of insulating 
materials caused by gamma radiation would be indexed under 
Deterioration, Insulating materials, and Gamma rays, not under, 
say, Radiation effects, Insulating materials and Nuclear 
radiation. At the same time the indexer was free to include 
other more general terms if he thought them likely to be useful. 

As mentioned previously, in certain cases pos ting-up to 
generic terms was adopted. Those were additional to the more 
specific terms arising from the document itself. Pos ting-up 
was, in fact, adopted largely because of problems caused by 
the limitations on the size of profiles. 

Posting up Terms 

Semiconductor devices, Semiconductor materials, Oscillators, 
Amplification, Control systems, Communication systems, Modula- 
tion Lasers, Gases, Metals, Aerials, Filters, Alkali Halides, 
Rare Earths, Pulse modulation, Microwaves, Alloys, Radar, 
Waveguides . 

Exhaustivity of Indexing 

The number of descriptors to be assigned per document was 
not laid down, but was left for the indexer to decide. 
However, in the light of experience during the initial stages 
of the Investigation it was generally agreed that the significant 
indexable concepts in the average document were not likely to 
require more than 10 descriptors. In fact, as will be seen 
from Figure 3* the average number of descriptors assigned 
per document was rather less than this. In the period from 
week Oil to 025 inclusive an average of approximately eight 
descriptors per document were assigned. These figures in all 
cases exclude author descriptors. 

At week 026 a change had to be made in indexing practice owing 
to a very large increase in the number of documents to be 
indexed. This arose as a result of the ending of various 
industrial disputes in the USA which had held up supplies of 
journals. To cope with this sudden "inflow of documents it 
was necessary to restrict the number of index terms assigned 
per document. This was less to save the time of the indcxers 
than to allow increased throughput by the tape typist. (The 
capacity of the tape typewriter was at all times a serious 
limiting factor in the system). 

To increase throughput the indexcrs were asked to restrict 
the number of terms per document to not more than five except 
in cases where this would result in obviousJy inadequate: * 
indexing. The five terms did not incJ ude any additional tonus 
required to fulfill the requirements of posttng-up. 
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The in CI ok of documcnLs continued Tor a considerable time 
at. a much higher love I Mian had boon expected so that what 
had been adopted as a temporary emergency measure persisted 
for approximately throo-and-a-haLf months. At the end of 
this lime the restriction was lirted but* the average number 
of tex^ms assigned therearter never reached the previous level. 
The reason for this was probably partly, by the now, ingrained 
habit* of reduced indexing and partly refinement of the 
indexing as a result of system performance analysis. 

The average number of index terms assigned, per document over 
the course of the operational service is shown in Figure 3. 

Descriptor file printout 

With prolonged use of the thesaurus the need for indexers to 
look up terms in the card file diminished. However because 
the computer validation check on documents input to it 
insisted on use of the exact form of word (i.e. would not 
accept singular for plural or vice versa) it was necessary 
in many cases for the indexers to check the form required. 

To facilitate this checking each indexer was provided with 
a reguLarly updated print-out; of the Descriptor File used 
by the computer for the validation check (see Appendix 13A). 
This undoubtedly helped to reduce the incidence of 
erroneous forms of descriptors and saved time. At the same 
time it led to a tendency to use it when consultation of the 
full thesaurus would have been more appropriate. For example 
an indexer faced by the concept 'Modulation 1 in a document- 
might check the Descriptor File printout to ensure that this 
term was present whereas consultation of the thesaurus would 
have shown the Narrower Term 'Elec trooptic Modulation 1 which 
might have been more appropriate. 

This leads to consideration of the value of the thesaurus 
■in the SDI system. 



Without doubt the development and maintenance of thesaurus 
is a very large task requiring considerable experience on 
the part of the person responsible and a great deal of 
associated clerical work. The most valuable assistance it 
provides in return is a record, of the decisions made in 
coding particular concepts, e.g. "Colour television receivers 
Usf ; Colour television + Television receivers", since without 
this record a variety of variant codings could be applied, 
e.g. Colour + Television + Receivers, Co l.our receivers + 
Television etc. This record allows indexing of documents 
and compilation of search profiles to use the same coding 
and thus increases the chance of retrieval. 
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In other respects the thesaurus is perhaps less useful than 
as sometimes supposed. Control of synonyms, which is one of 
lr%lZ U C vjrtues » of course important, but the incidence 
of true synonyms xn science and technology is low, particularly 
when dealing with a relatively restricted field. wLre two 
words exist to cover the same meaning it is often because 
they are not truly synonymous over the full range of applica- 
tions, even if lt is only that an engineer will tend to use one 
form and the pure scientist the other. Where true synonyms 

tL^ C T\ e L aerialS and antennas » fact is usually welx 

Known to both users and operators of the system. 

Recording of relationships is. also a useful function of the 
thesaurus, showing Broader Terms, Narrower Terms and reJated 
Terms However, many of the BT and NT relationships are self- 
evident and many of the RTs are unsatisfactory in that the 
relationship is ill-defined. 

Many of the lead-in entries are almost redundant, being of 
the type, 'Filters optical, Use Optical filters'. The type 
of lead-m which is needed is 'High field microwave oscilla- 
tions xn bulk semiconductor materials' Use 'Gunn effect' 
These are, of course, much more difficult to include and 'use. 

All this is not to dispute the need for some of this informa- 
°u L av ailable to users of the system but to cast doubt 
on whether much of the information contained in thesauri is 
not redundant. This takes considerably time and effort to 
add to the thesaurus but serves often only to mask the much 
smaller number of entries which carry really significant 
information. 

It seems possible therefore that more examination is needed of 
the real role played by the thesaurus in document indexing and 
search-profile construction and of the form in which it can 
most effxciently and economically meet these requirements. 
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Figure 5 . 

Average number of index terms assigned .per document during 
the weekly operational service, period 
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SDI DOCUMENT INDEXING WORK SHEET 



FIGURE k. 
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Descriptor Sheet 



FIGURE 5. 



::di PocuMEii r ihtus? form 



Spaces mv.st not be punched except where a space symbol (V) 
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Nothing outside this border is to w punched 



• Title 

Authors 

Journal 
Citation 
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Authors 
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Chapter 1^ 



PHOnLEMS OF PRINTING NOT! FIG AT I.ON CARDS ANT") 
ASSESSING NUMBERS REQUIRED FOR EACH DOCUMENT 



In any SDI system it is desirable, Iron, the standpoint of 
cost, that the 'utilisation rate' per item of information 
input should be high i.e. that the average number of relevant 
notilications produced per document should be large. 

tLTl SamG * ±me th ° util± sation rate may affect the method 
used for producing the document notifications. For example 
in a system operating with a small data base within a 
restricted subject field, the number of notifications per 

It n a *5° CU ?! nt ^ b ° VGry hich ' In such circumstances 

it may be thought more economical to produce multiple copies 

other thanY? f ° r dcs »* tch to users oy means 

orner tnan a line printer. 

In e the n SD? C S ti0 ? S J ^ din ? t0 US ° ° r <*«P"ca*od notifications 
oCtout eatl ° n ^eluded the cost of line printer 

output, but the major consideration was legibility and 
appearance of the notifications. Tt was thought that cards 
printed using typewriter upper and Lower case would be much 
more attractive to users than computer upper-case printout 

notijLaSonr^ ^"h 5 ,10W ° VOr CaUS ° Problems, since the 

Jrom in h? USer had t0 be individually selected 

from, in this case, a set of approximately 300 cards. This 

c:rr?ravo1a y s a i ti T C '- COn J Umilie taSk rm*C "nSderSlo 
care to avoid selection of incorrect cards. The error rito 

was in fact surprisingly low but this can only be a^rL^u^ed 
to the great care taken by the clerk responsible! attrLhuttd 

However the chief problem in use of duplicated no t.l f icat Lon s 
even m a system where the number of documents input to each' 
run is relatively small, j. s that of determining how many 
notification cards to produce for each document. 
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Obviously this is most easily done after the matching ruti 
has been completed, when, if the computer is producing 
adequate statistics, the number of notifications for each 
document will be known. 

Such statistics wore not available from the SD1 program 
suite and therefore this option was not open. However 
even where this facility is available the Information is 
really required earlier than the end of the matching run. 
To ensure that notifications are as current as possible it 
is essential that the duplication of notification cards and 
the computer matching should be carried out in parallel 
rather than serially. In the SDI Investigation, production 
of the notification cards could in fact commence before 
despatch of the document tapes to the computer and in general 
all cards were ready by the time the matching output was 
returned. 

In the early weeks of the investigation, counts were made of 
the number of notifications per document to decide how many 
notification cards to produce. 

The results are shown in Figures 1 and 2. 

It can be seen that the distribution has a considerable tail 
and that some few documents feature in a hundred or more 
notifications. It is fairly simple to work out the number 
of cards that must be produced to cover a given percentage 
of cases but is is not a simple matter to print extra cards 
for particular documents and the complication caused in 
selection and despatch by waiting .for extra cards to be 
produced is very considerable. 

An examination was therefore made of the documents which 
were found to require more than 70 notification cards to 
discover whether they were on particular subjects which couJd 
be given special treatment. The distribution of such 
documents in weeks 0ll-0l'» is shown in Figure 3. 

The majority of these documents related to 'microwave semi- 
conductor devices' and to 'integrated circuits' both of which 
subjects were of interest to a large proportion of users. 
Since it was obviously wasteful to print enough notification 
cards for each document to cover all expected demands it was 
decided to print 50 cards for each document and 90 for those 
in the popular subjects. Even this solution, which was stiU 
wasteful in cards, did not meet the problem entirely and 
significant trouble was encountered "in running off additional 
copies of cards. • 




It is always likely that in any SDI service the frequency 
of notifications of individual documents will have a 
distribution similar to Figures 1 and 2. 

Apart from the problems of selecting individual sets of cards 
there will always bo difficulties in dealing with the long 
tail of highly used documents. Short of accepting the delay 
involved in waiting for statistics of document notification 
frequency, it is likely that, even with a fairly restricted 
data base, the extra computer print-out cost invoJved in 
printing individual notifications for each user will have to 
be accepted. 
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Chapter 1 5 
RECALL AND PRECISION FAILURE 

An examination of the reasons why relevant documents were 
not selected by prof Lies (Recall failure) and why irrelevant 
items were selected (Precision failure) was made using data 
from weeks 01k and 023. The material examined lor Precision 
faxlure comprised documents notified to a sai-plc of h8 
users xn these two weeks, but marked bv them as X (not rele- 
vant ) . 

In the study of Recall failure it was decided to analyse 
those docu icnfcs in week 023 which the user's profile had 
failed to select but which he had marked as K1 (highly 
relevant) on the sample list of documents submitted to him. 
The number of partially relevant (R2) documents marked was 
large and would have taken too long to analyse if the same- 
sized sample of users had been retained. 

For the documents of week OTt, users had not been asked 
to differentiate between highly relevant and partially 
relevant documents and , there fore , only a single relevance 
marking was available. To reduce the number of documents 
requiring analysis only a sample of the 01't documents was 
included. The results of the study were calculated using 
both the mixture of 023 and OUi documents, and highly 
relevant 023 documents alone. 

Failure Categories 

A number of possible reasons for failure wens isolated, 
classified primarily by whether the failure was one of 
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derail or Pn-csion, and secondly l,y whether t I. was causc'cl 
by .lcric:,,.,ic:i«;s in Uic search profile, in Llio document 

indexing, or- in the thesaurus.* The catur K ories wore as 
Follows: 

Recall Fa i lures 

a) Due Lo proI'Llo 

RPF 0 Subject or document Is considered to be outside 
the scope oP the user's stated information requirement 
Cannot truly be considered a failure of the system. 

RPF 1 Document obviously Falls within the user's 
stated requirements. Search proiile is dercctlvc'. 

RPF 2 Document de^.ls wLth a subject which mi^ht 
reasonably, particularly with the benefit of hindsight 
have been assumed to fall within user's requirements. 



RPF 



J£ Subject of document Tails within the user's 



stated requirements but the particular requirement 
i.s so general or va^uc that ifc was deliberately 
omitted In view of likely adverse effect on Precision, 



RPF 



Jl Other causes of failure, e.^. lo^lc statement 



of the search prorile i.s too restrictive to select 
the document. 

b ) Documenting i ndexln/: Fa ilui-es 

RIF 1 failure to Index a concept which Ls obviously 
present and Important In the Mocunient. 

RIF 2 Failure to index a concept which can be seen 
to be present when studied In the lL{:Iit oi* a 
particular user's Interests. 

RrF 3 Failure* to aipply automatic 1 pos tin/;-up ' 

to a Broader Term as laid down for certain terms. 

RIF h Other Indexing failures. 

c ) Thesaurus 



RTF The particular index term assigned to the 
document was added to the Thesaurus after the 
last upda tin/; of the- pro Pi It. 



I'rm: j s i on fa i I tu-ttx . 

■'») Dim to prof i I < ? 

PPF 0 document appears to moot Lho user's stated 

requirements. Further co ui:i cation with user is 

needed to discover his reason for marking it non- 
relevant. 

PPF j Document retrieved by false co-o rdina t ion 
ae. 'combination of terms specified in profile h^-.s 
unexpectedly soJ.oc tod irrelevant document. 

PPF 2 Deliberate loosening of profile, intended 
to secure good Renal J performance, has caused 
expected Precision failure. 

p PF 3 Other profile faljure. 

b) Document indexing 

PPF 0 Poor indexing eg. misunderstanding of subject 
of document; use of wrong index term. 

PPF_L Over-zealous indexing. Xndoxor has included 
minor aspects of tlie document in his indexing. 

PIF 2 Automatic postLng-up to more general term 

has caused retrieval of an unwanted fringe document. 

c) Thesaurus 
PTF 

Index term available is too wide and does not allow 
distinction to be made between relevant and irrelevant 
concepts. 

Anal^ sis 

r n t°l al °f f l9 ? i dLCuniG » t s IVom weeks 0 Ml and 023 were analvsed 
for Recall failure and ! 3 J for Precision failure. The sample 
of Rl documents for week 023 contained 59 for Recall failure 
analysis and 93 for Precision failure analysis. 

The results are shown in Figs. 1 and 2. 

An alternative method of analysing the results was used which 

involved looking at the failures and assigning the .« the 

following categories: 



l) Failure of user-system interact i 



on 
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2) Operaror error or de.f Lc Lency . 



3) Swings and roundabouts effect caused by the inverse 
relationship between Recall and Precision. 

'») Thesaurus/system 

Each of the RPF, PPF etc. categories was examined" and 
assigned to whi.cli.ivor of the above- lour categories seemed 

Results and Conclusions 

The relatively small number of failures analysed Li, the 
study and the fact that they are drawn from two weeks means 
tvnL»?V a M° S eaCh tyPC ° f ' faii "rc >»ay not be entirely 

seen on rY, Sy f C '" as a ^lo. However, Lt can be clea'rlv 
seen from figures 1 and 2 that failure in both Recall and 

Sl C Si? n H S * OV ? rWhel i"i ,,,rly assocLaU ^ "itl. Lhe search profiles. 
?t*£h.«- , a ^" OSL 10 P erccnt or «ocaJl failures could be 
attrxbutocl to doeii;,„.,its nnrkorl by the user relevant 
be.ng considered outsido tl IO scope of the user's InformnMon 
acquitments, presuppose, a considerable ,np be tu,e„ , h. "s " r • « 
Mew ol his requirements, the subjects of the documents ho 
Z rt as |J rele ^';'. and Lhc SDI raff's idea of his requirements. 
On the other hand Lt ls not likely that the problem arises 
simply through too narrow an Interpretation bv the SUE staff 
of the user's requirements. Rather It must b'e considered 
a general fault of the system. 

The philosophy adopted in profile compilation was to treat 
the user's information requirements- as a whole, rather than 
to ask hi... to restrict himself to one or t^o h Lghlv specific 
aspects. It was felt that the sort of specific search - ' 
request put forward by a user to a retrospective search inr 
system may not be the form to represent his continuing ne£d 
for current-awareness information. If one assumes that his 
current-awareness needs will be more general than his 
retrospective search request and will have a core of highly 
specific interests surrounded by outer layers of more general 
and less-defmed interests, then a problem arises - what 
port.on of these Interests should ; ,„ sin profile ;,l.to„.pi 
to cover. 1 

One common approach is to concentrate on Lhe hirhLy spec i I* it- 
core mtcrests. This tends to produce a well-do fi .,«<! search 
Profile of the same type as the retrospective, search request. 
Hiis of cour-.i works well for people e.g. research scientists 
who are working on a highly specific area but may be less 
sattsfactorv for the engineer who may need a variety of 



in format ion on d i-f f oron I aspects of a number of fields. 
The basic problem Ls that it is very difficult to arrive 
at the pure statement of the user's roaJ needs since he 
normally has to be guided on the method of presenting his 
needs to the system. The examples provided to film force 
him to adopt a particular view of his requirements which 
may or may not reflect his own view. Thus the Information 
requirements of users of the IN SPEC SDI service wore of an 
entirely different type from those expressed by, for example 
the users of the UKCIS service (ignoring the obvious 
difference of subject). 

It could be argued that the information* requirements 
statement given by INSPEC users reflected more nearly their 
lull needs, but this can only bo considered a- virtue whore 
th-> system servos these needs 45 at si sfac tor i 1 y . 

A consequence of Llie attempt to cover the ful L range of 
informal iou requirements of a user is that document marked 
relevant by a user must be cons i.' crcd relevant to his 
information requirements, and therefore, failure to retrieve 
it must be considered a real failure which although it can 
be explained by the fact that the particular concept was not 
mentioned in his statement of information requirements, can- 
not be ignored in assosing the performance of a system which 
aims to cover his complete range of interests. 

It is clear that a system which attempts to serve the full 
needs of a user, ranging from the very specific core interests 
to the general fringe interests, and tries to adapt to 
changes in these interests, must allow much more direct and 
rapid interaction between tha user and the system that was 
possible in the Investigation or in most current SDI systems. 

The compromise in most cases is to settle for some narrower 
section of the user's requirements which can bo more casily 
dealt with by present systems. The result- is that most 
SDI systems operate in the same way as retrospective socirch 
systems, the only difference being that the search is repeated 
regularly over the relatively small number of documents boirir 
added each week to the 'ata base. 

Most of the failures of the INSPEC SI)} system appear to follow- 
directly from this attempt to cover the wide range of a 
user's requirements in a singJo pro«*i|o. The biggest s j„ r | 0 
cause of Recall failure is NPP 0 - documents considered Lo 
be outside the stated roqn i remonts , while the next most iniporl.;mi. , 
RPF 1, is failure to retrieve documents which might .just 
conceivabJy be considered Lo la I I within the scope of Uw 
user's requirements. Only some: five percent of f.ii linos wore 
considered to relate to documents which obviously coincided 
with the user's stated requirements. 
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On the ronton side, Uin IW 2 failures may bo attributed 
to tho factors rolatod to Uio fjoaeral nature »r tho i n for- 
mat, or, rm„,lro«otit8 of usors and tho fact that relevance 

ronoraTw ;1 ""I ts " lt '» lnLlnn °«" P«»T«*..co wore 

Lo o.alJy ,Jo„„ », y usor , s wlu ,out direct roforence to their 
s t.i t ed requ i roinori t s • 

In sum, therefore, it would appear. that tho vast majority of 
failures of the system derive from tho general nature of the 
users information requirements and the- problem of obtaining 
consistent relevance assessment in such circumstances * 

Other types of failure are by comparison small in number. 

of ShfrivS! ° C ° UrSO ' nCan 1:hat Lh ° co ^espondLn G aspects 
of this system are necessarily better than in othor systems 
since the study is concerned only with the major cause of 
failure in each case and the profile aspects may cover 
instances of other types of failure. 



2**-± Recall and Precision fai lure analy sis for samnl „ 
ol docu ments for weeks 01 k and 7)2g — 
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Fi fr 2 Recall and Precis ion failure analysis for sample 
of documents input to week 023 



Recall 
failure 
catp fa rv 


No* nf' 

A> W ft U JL 

articles 


yo o I 

total 


Pre cis ion 

failure 

category 


No . of 
articles 


# of 
total 


RPP .0 






JrJPF 0 


17 


18 


1 






1 


6 


7 


2 






2 


27 


29 




10 






a 29.5 


32 

* 


k 












RIF 1 


1 


2 


PIF 1 


5 


6 


2 


1 


2 


2 


2.5 


3 


3 








3 


3 














RTF 


0.5 


1 


PIT 


3 




TOTAL 


59 


101 


TOTAL 


93 


101 
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Chapter 16 



PRECISION PERFORMANCE OF THE SDI SERVICE 



Throughout the SDI Investigation, users were "asked to return 
relevance assessments for all documents notified to them. 
They did so by marking one copy of the notd xcation form with 
a 1, 2, or X .against each document number and returning it 
to INSPEC. 

The relevance categories were assigned the following meanings 

1 - Highly relevant i.e. documents such that the user 

would wish his profile to select all similar items 

2 - partially relevant i.e. documents of interest to the 

user but not of such direct importance that he would be 
concerned if a proportion were not selected by his 
profile ^ 

X - not relevant i.e. documents of so little relevance or 
interest, that the user would not wish his profile to 
select them. 

It can be argfrcd that three categories is either too many or 
too few. A number of users from time to time urged the need 
for finer distinction. They were told that there was no 
objection to their adopting a greater number of relevance 
classes provided only that they explained the basis of their 
classification so that proper use could be made of the 
assessments in profile analysis and modification. No user 
in fact took up this offer. 

On the other hand very few users made the case for a simple 
relevant/not relevant cJ assif ication possibly because they 
could in any case adopt this for themselves. 

Some users in fact appear to -have used fch« rr l.ovancc 2 
category so little that they were using a yes/no assessment. 



In general the three-category system appears to have been 
satisfactory for the purposes of the Investigation. Apart 
from its use in determining the Precision performance of a 
given profile or the system as a whole, it was also used in 
profile modification. In general the aim of profile 
modification was to maximise '.the number of relevance 1 
documents and to minimise the number of not-relevant documents. 
The relevance 2 documents were treated as largely neutral 
items whoso number was to be maximised as far as was consistent 
with the preceding major aim. 

There is of course a case for the two-category assessment 
since it can be argued that a user should^ be able to decide 
whether he is interested in a given document or not. However 
this yes/no assessment can cause problems for the staff who 
may have to reconcile two ^apparently contradictory assessments 
on similar documents. 

A very high proportion of users regularly sent in relevance 
returns. This was made easy for them by provision of a 
second copy of the form and a reply-paid envelope. In 
addition the importance of the regular feedback was stressed 
to users and if necessary the ultimate sanction of withholding 
the service could be imposed. However, taking all these factors 
into account, the percentage of users sending in returns each 
week is impressively high. 

The figures are shown for some sample weeks in Figure 1. j'rtm 
which it can be soon that the weekly average return was in 
excess of 90 'percent. 

* . 

Precision figures, were calculated in two ways:- 

a) Total relevant documents i.e. Relevance I and 
Relevance 2 doeurr its combined, expressed as 

a percentage of total documents sent. This is 
sometimes expressed as Precision Rl/2 or Rl/2 
Precision . 

b) Relevance 1 documents only, expressed as a per- 
centage of" total documents sent. This \s expressed 
as Precision/ Rl or Rl Precision. 

Performance figures were recorded each week for each profile 
and in general the Rl/2 Precision figure was the one adopted. 
However it became apparent (J|at for some users the relevance 
2 documents were of relatively minor in teres!, and a good Rl/2 
Precision performance figure could hide a very unsatisfactory 
profile. From week 035 onwards, therefore, both Rl and Rl/2 
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precision figures were recorded. This is not to say that 
previously the figures were not available for analysis but 
merely that they had not been set down in a form easily used 
for profile monitoring. , 

It had been intended originally to input the raw performance 
figures for each profile to the computer and to obtain the 
necessary performance statistics by this means. 

However as this part of the SD1 programs remained in a very 
rudimentary and untested state, the task of punching and 
inputting this data was not carried out as it was not 
commensurate with the small expected benefit. The performance 
figures for the system appear in several ways. As mentioned 
earlier there are two basic sets of figures .relating to relevance 
1 and 2 documents combined and to relevance 1 documents only. 
For the reasons mentioned earlier, while the Rl/2 figures are" 
given for the whole of the service period (weeks 011-066) the 
Rl figures are given only for the period 035-066. 

In each case the figures are calculated in two ways which, 
lollowmg Cleverdon, we have called 'Average of numbers' and 
'Average of ratios' respectively. 

Each gives a slightly different picture of system performance. 
Average of numbers reflects the performance of the system as 
though the individual user requirements were amalgamated- to 
form the general requirements of one generalised user and each 
document retrieved carries equal weight in calculating" the 
•Precision figure. Using average of ratios the performance 
figure represents the sum of the individual performance of 
each profile and thus perhaps reflects more closely the average 
satisfaction rate of the service. 

Where the average number of documents retrieved by each profile 
39 approximately the same , difference between the two measures 
is small. However, where there is a mix of users requiring 
high precision with profiles producing fev but large 'y 
relevant documents, and other requiring high recall with * 
profio.es producing many documents a large proportion of 
which are non-relevant, the difference in the performance 
figures calculated by the two methods may be large. 

In terms of examining the (if feet of various parameters on the 
system, the average of numbers is preferred; but for assess- 
ing the success of the service in meeting user needs, uvornro 
of ratios is considered the best yardstick. 

The precision performance of the system, based on Rl/2 
documents, is -ubulated in Figures 2a and 2b for both average 
of numbers and average of ratios. Craphs of these vaJno 
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shown for period 011-038 and 039-066 in Figures 3-6 • Similar 
values based on relevance 1 documents only are shown in 
Figures 7a and' 7b and "plotted in Figures 8-9 for weeks 039-066, 

In each case the graph is plotted for all (approximately 576) 
profiles, for those profiles (appx^oximately 105 ) which 
remained unmodified throughout the period 011-066, and for 
the approximate 450 profiles which were modified during 
this period. It will be seen that the unmodified profiler 
perform significantly better than the others. Tliis is not 
particularly surprising since the reason they remained 
unmodified is, in the majority of cases, due to their 
satisfactory performance. At the same time, since the 
superior performance is largely maintained in spite of major 
modification to the other profiles it would appear that 
the superior performance is due to some intrinsic quality 
of the original requirements of xhn user. Nevertheless 
there does appear to be a tendency for the margin of 
superiority to be eroded during the course of the investigation, 
with the performance of the unmodified profiles gradually 
dropping towards the somewhat improved level of the modified' 
ones. This fall could well be explained by the user*s real 
requirements moving further away during this time from those 
stated at the beginning of the investigation. 

In Figures 10-12 are plotted the performance figures Rl 
and Rl/2 for all profiles. These show that in the majority 
of cases the pattern of Rl precision is closely related to 
that for Rl/2 precision. 

The single precision figure of course shows only part of 
the picture andit is of interest to know the distribution 
of individual profile performance. These arc given in a 
series of tables and graphs showing Rl and Rl/2 precision 
based on average of numbers and average of ratios as follows : - 

Figures 13-15* 

Figures 16-18 

Figures 19-21 

Figures 22-24 

Figures 25-27 
Figures 28-32 

precision. 



All profiles, Rl/2 precision 
Unmodified profiles, Rl/2 precision 
University profiles, Rl/2 precision 
Government establishment profiles, 

Rl/2 precision 
Industrial profiles, Ri/2 precision 
All profiles and unmodified prof. lies 
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Figure 1. Users receiving notif icattions and returning assessment!, . 
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Figure 7a. Precision performance RJ of SDI profiles over the 
period 035-066 - all profiles . 



Week -No. Average of Nos. Stand. Dev. Average of ratios Stand. Dev. 



035 


35.4 


26.7 


35.6 


31 .7 


036 


35-3 


27. 1 


32.7 


j\J • y 


037 


36.5 


27.5 


33.7 


^1 P 


038 


33^3 


25.0 


35.6 


ft 


TOTAL 


35.2 


21.9 


3k. 1 


24-7 




34.7 


' 26.8 


34.0 


30 • 5 


o4o 


33.9 


26. 1 


33. 3 


2Q 7 


o4i 


30.9 


23.7 


31 -7 


29 .k 


042 


32.7 


26.4 


30*8 


30 2 


TOTAL 


33-1 




32.8 


23. 2 


043 


33.6 


29.9 


32. 3 


34 Q 


044 


34.7 


27.7 


J2.7 


3^ 1 


045 


33.2 


25.9 


30.9 


3o i 


046 


35.5 


26.7 


34.3 


31 

J> 1 • 1 


TOTAL 


34.3 


21.7 


32.8 


PU P 


047 


32.7 


27.0 


29 .4 


**P 7 


048 


37.4 


28. 1 


34.8 




049 


27,7 




2o. 1 


32.0 


050 


3K2 


26.8 


32-3 




TOTAL 


32.2 


22.3 


31.2 


25.0 


051 


30.6 


27.5 


28.3 


31.7 


O52 


30.5 


26.5 


28.4 


30.4 


053 


32.3 


28.6 


33.4 


34.7 


055 


34.8 


26.2 


33,2 


30.8 


TOTAL 


32.2 


22.3 


30.4 


24.0 
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Fi t ?ure ?a. (Contd) 



Week No, 



Average of Nos. Stand.Dev. Average of ratios Stand. Dev. 



055 
056 

O57 
058 
TOTAL 

059 
060 
061 
062 

TOTAL 
O63 
064 
066 
066 

TOTAL 

GRAND 
TOTAL 



29*9 

29. h 

30.5 

28. i 

29.5 
32.6 

35.1 
31.7 
29.9 
32.5 
29.2- 

33.3 
31 .8 
28.1 
30.9 

32,5 



24.0 
24.3 

23.5 
23.6 
20.2 

23.7 
26.6 

25.3 
25.1 
20.6 
24.6 
26.2 
26.1 

25.5 
21 .4 

19.0 



28.7 
25.7 
30.7 
27.7 
29. -1 
32.2 
32.8 
30.8 
28.8 
31.2 
28.1 
30.2 
27.8 
28.0 
29^3 

31.5 



27.3 
27. *t 
28.9 
28.3 
23.6 
29.4 
29.7 
31 .1 
31.3 
22.6 

29.5 
29.1 
30.1 
29.7 
23.7 

19.2 
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Figure 7To . PRECISION PERFORMANCE R1 OP SDI . PROFILES .OVER THS 
PERIOD 035-066 •- I^aifjgd profiles 
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:RIC 



Week No. 


AvG "Pa P*^ of* Mftc 




Average of ratios 


Stand. Dev. 


035 


i»2.2 


' 27.5 ■ 


- 

■ kl.8 


30.5 


O36 


^3.1 


29.0 


36.3 


32.2 


037 


kk.6 


30.0 


1*2.1 


33^9 


O36 


38.2 


26.6 


38.9. 


30.9 


TOTAL 


**2.1 


2*U2. 


39.5 


25.7 


O39 


37.9 


28.8 


36.3 


30.6 


0^0 


37.0 


26.8 


36.6 


28.8 


0^1 


36. k 


25.6 


3,6.** 


30.2 


0h2 


kO.5 


29.it 


38.2 


31.9 


TOTAL 


37.7 


23.6 


36.9 


23.2 




^0.7 


3P*2 
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36.5 


okk 


40-. 1 


30. 0 ■ 


3^.0 


32.7 ... 
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27. 1 
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30.7 




39.1 


28. 7 
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32 . k 


TOTAL 


39.0 


23.7 


35 = 3 


23.9 
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30.6 
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3^.9 
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29.3 


^2.5 


32.3 


049 


29.0 


26.6 


25.2 


30.9 


050 


36.O 


31 .3 


36.1 


32.7 


TOTAL 


36.9 


25.2 


35.2 


26.3 


051 


36.7 


31.6 


35-7 


35. * 
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3^.7 


30.3 


28.2 


29.9 


053 


36.7 


31 . i 


37-7 


35.8 


055 


39.6 


26.8 


35-7 


28.9 


TOTAL 


36.9 


25. 4 


3^.3 


24.2 


055 


35.1 


27.3 


33*1 


27.9 


056 


3**.8 


29.1 ■« 


28.9 


29.2 


057 


36.3 


26.3 


37.6 


30.2 
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32.5 


28.6 


33-3 


30.8 


TOTAL 


3^.7 - 


2k. 9 


35.3 


27.4 


059 


38.2 


26.0 


39.1 


29.6 


060 


39.6 


27.8 


35.5 


28.3 


061 


38.0 


■ 28.5 


32. k 


31.5 


062 


35.9 


29.2 


35.9 


33.3 


TOTAL 


38.0 


23.7 


36.3 


2k. k 



FIGURE T3. PRECISION PERFORMANCE OF ALL PROFILES OVER THE 

PERIOD VEEKS 011-066 (RELEVANCE 1 AND 2 DOCUMENTS 
COMBINED). 



# NO. OF PROFILES 



PRECISION 


AVERAGE OF NUMBERS 
No. 1o 


AVERAGE OF RATIOS 
No... % 


0 

1-10 
11-20 
21-30 
31-40 
41-50 
51-60 
61-70 

71-80 
81-90 
91-100 


2 0.3 

5 0.9, 
10 1.7 
26 k.5 
51 8.8 
93 16.0 
122 21.0 
111 19.1 

88 k 1 

68 11.7 


2 0.3 

5 0.9 
15 2.6 . 
21 3.6 
48 8.3 
102 17.5 
112 19.3 
120 20.6 
So 15.1 
63 10.8 


Total 


576 100 


576 100 
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Figures 16. PRECISION PERFORMANCE OF 104 UNMODIFIED PROFIL ES 
OVER THE PERIOD WEEKS 011-066 (RELEVANCE 1 AND 2 
DOCUMENTS COMBINED.) 



PRECISION 



NO. OF PROFILES 



AVERAGE OF NUMBERS 
No. % 



AVERAGE OF RATIOS 
No. % 



0 




1-10 




1 1-20 




21-30 




31-40 


1 


41-50 


5 


51-60 


13 


61-70 


20 


71-80 


21 


81-90 


24 


91-100 


- 20 


Total 


104 



1 .0 
4.8 

12.5 
19,2 
20.2 
23. 1 
19.2 

100 



1 

7 

15 
19 
20 

25 
17 
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1 .0 

6.7 
14.4 
18.3 
19.2 
24.0 

16.3 

100 
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Figure 19. PRECISION PERFORMANCE OF SDI PROFILES OVER THE 

PERIOD WEEKS 011-066 (RELEVANCE 1 AND 2 DOCUMENTS 
COMBINED) - UNIVERSITIES 



¥> NO. OF PROFILES 



PRECISION 


AVERAGE 




AVERAGE 


- 

OF RATIOS 




No. 


■7° 


no • 


Jo 
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1 1-20 
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28 


16.6 
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19.5 


30 


17.8 
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41 
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40 
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15.2 
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13.6 
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13.6 
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168 
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FIGURE 22. 



PRECISION PERFORMANCE OF SDI PROFILES OVER THE 
PERIOD W EEKS 01 1-066 (RELEVANCE 1 AND 2 DOCUMENTS 
COMBINED) - GOVERNMENT ESTABLISHMENTS. 



PRECISION 



NO. OF PROFILES 



AVERAGE OF NUMBERS 
No. # 



0 
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1-10 






11-20 


1 
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0.6 


31-^0 


11 


6.0 


41-50 


15 


8.2 




29 


15.8. 


61-70 


44 


24.0 


71-80 


33 


18,0 


81-90 


21 


11.5 


91-100 


26 


14.2 



AVERAGE OF RATIOS 
No. # 



Tot?l 



I83 
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2.7 
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FIGURE 25. PRECISION PERFORMANCE OF SDI PROFILES OVER THE 

PERIOD WEEKS 011-066 (RELEVANCE 1 AND 2 DOCUMEN TS 
COMBINED) - INDUSTRIAL FIRMS. 

1 



NO. OF PROFILES 



PRECISION 


AVERAGE 


OF NUMBERS 


AVERAGE 


OF RATIOS 




No. 




No. 




U 


2 


1 . 1 


2 


1.1 


1-10 










11-20 


2 


1.1 


2 


1.1 


21-30 


4 


2.2 


5 


2.6 


31-40 


3 


1.6 


4 


2.2 


41-50 


19 


10.3 


20 


10.8 


51-60 


35 


18.9 


37 


20.1 


61-70 


39 


21 .2 


34 


18.5 


71-80. 


32 


17.4 


• 36 


19.6 


81-90 


33 


17.9 


30 


16.3 


qi 1 nn 


15 


8. 1 


14 


7.6 


Total 


184 


100 


184 


100 
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FIGURE 28. PRECISION PERFORMANCE OF SDI PROFTLES OVER THR 

PERIOD WEEKS 035-066 (RELEVANCE 1 DOCUMENTS ONLY) 
ALL PROFILES AND 142 UNMODIFIED PROFILES - 
AVERAGE OF NUMBERS. 



°h NO. OF PROFILES 



PRECISION 


ALL 


PROFILES 


UNMODIFIED 


PROFILES 


• 


No. 


% 


No. 




0 


7 


1.3 


1 


0.7 


1-10 


55 


10. 1 


10 


7.0 


1 1-20 


115 


21.2 


23 


16.2 


21-30 


128 


23.6 


31 


21 .9 


31-40 


88 


16.3 


25 


17.6 


■ 41-50 


57 


10.5 


20 


14.1 


51-60 


42 


7.8 


13 


9.1 


61-70 


27 


5.0 


8 




71-80 


16 


3.0 


7 


4.9 


81-90 


5 


1.0 


2 


1.4 


q 1 — 1 on 


3 


0.5 


2 


1.4 


Total 


5^3 


100 


142 


100 
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